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Preface 


In  the  early  years  of  Advanced  Information  Services  Inc.  (AIS),  we  were  not  profitable  be¬ 
cause  our  projects  were  not  predictable.  Most  of  our  projects  were  on  fixed  cost  contracts  and 
experienced  schedule  and  budget  overruns.  Although  quality  was  not  measurable,  indications 
of  our  quality  problems  were  the  significant  amount  of  rework  necessary  at  the  end  of  a  proj¬ 
ect  and  the  attendant  customer  dissatisfaction.  People  worked  long  hours,  and  heroic  efforts 
were  needed  to  complete  projects. 

In  1992,  our  Chief  Executive  Officer  (CEO),  Girish  Seshagiri,  attended  a  conference  on  con¬ 
tinuous  process  improvement  based  on  Watts  Humphrey’s  Managing  the  Software  Process 
[Humphrey  89].  Girish  returned  from  the  conference  convinced  that  process  improvement 
would  solve  our  problems,  and  he  communicated  his  vision  to  everyone  in  the  organization. 
The  onset  of  the  initiative  was  for  us  to  base  our  software  process  improvement  effort  on  the 
Capability  Maturity  Model®  (CMM®).  In  1995,  that  initiative  evolved  into  using  the  Personal 
Software  ProcessSM  (PSPSM)  along  with  the  CMM. 

Over  the  lifetime  of  the  initiative,  the  AIS  Development  Group  has  transformed  its  ad  hoc 
development  into  practices  that  are  process  oriented  and  disciplined.  Our  average  project 
schedule  overrun  has  been  reduced  from  112%  to  5%,  and  our  average  budget  overrun  from 
87%  to  —4%.  We  now  average  one  defect  per  20,000  lines  of  code  in  delivered  products.  One- 
half  of  the  products  delivered  in  the  past  three  years  have  been  defect  free.  The  rewards  of 
these  schedule,  budget,  and  quality  successes  are  satisfied  customers  and  company  profitabil¬ 
ity. 

The  AIS  Development  Group  was  recognized  for  these  results  by  the  software  professional 
community  with  the  1999  IEEE  Computer  Society  Software  Process  Achievement  Award.  In 
addition,  AIS  was  recognized  by  the  business  community  with  a  1999  Blue  Chip  Enterprise 
Initiative  award  by  the  U.S.  Chamber  of  Commerce  and  Mass  Mutual. 

The  AIS  culture  has  changed.  Process  improvement  is  no  longer  something  that  people  need 
to  be  convinced  to  do;  it  is  part  of  the  day-to-day  work  effort  at  AIS.  Good  planning  and 
tracking  and  significant  reduction  in  rework  has  resulted  in  people  working  reasonable  hours 
with  higher  productivity  rates. 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
SM  Personal  Software  Process  and  PSP  are  service  marks  of  Carnegie  Mellon  University. 
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This  report  provides  a  brief  history  of  the  software  process  improvement  initiative  and  the 
results  achieved  thus  far.  AIS  will  continue  to  improve  because  of  the  strong  foundation  put 
in  place  by  the  leadership  of  Girish  Seshagiri  and  by  the  contributions  of  many  software  en¬ 
gineers  and  project  managers  over  the  years.  We  would  like  to  acknowledge  Watts  Humphrey 
whose  dedication  to  software  quality  has  been  an  inspiration  to  us.  The  methods  that  he  has 
developed  while  at  the  Software  Engineering  Institute  (SEI)  have  provided  the  foundation  for 
our  improvements. 
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Abstract 


The  Advanced  Information  Services  Inc.  (AIS)  Development  Group  was  motivated  by  busi¬ 
ness  needs  to  begin  its  Continuous  Process  Improvement  (CPI)  initiative  in  1992.  We  chose 
the  Software  Engineering  Institute  (SEI)  Capability  Maturity  Model®  (CMM®)  as  the  process 
maturity  framework  [Paulk  93]  and  the  Institute  of  Electrical  and  Electronics  Engineers 
(IEEE)  standards  as  the  guidelines  for  software  engineering. 

We  can  relate  the  changes  in  AIS  process  capability  to  three  different  eras.  The  first  was  the 
time  before  1992,  during  which  AIS  did  not  use  a  model  for  software  process  improvement. 
During  the  second  era,  1992  to  1995,  the  CMM  was  used.  Data  indicate  that  project  schedule 
and  effort  predictability  improved  with  the  introduction  of  the  CMM  framework.  The  third 
era  began  in  1995  when  AIS  piloted  the  Personal  Software  ProcessSM  (PSPSM).  With  the 
adoption  of  the  PSP,  data  reflect  an  increase  in  product  quality  and  improvements  in  produc¬ 
tivity  levels. 
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1  Background 


1.1  History 

Advanced  Information  Services  Inc.  (AIS)  is  an  independent  software  contracting  organiza¬ 
tion  that  operates  a  main  office  in  Peoria,  Ulinois,  a  satellite  office  in  the  Chicago  area,  and  a 
subsidiary  in  Chennai,  India.  The  business  segments  of  AIS  include  software  development, 
consulting,  software  process  training,  and  Internet  services.  AIS  began  developing  commer¬ 
cial  systems  in  1986.  Embedded  system  development  effort  began  in  1997.  The  AIS  Devel¬ 
opment  Group  maintains  a  staff  of  between  20  and  35  people. 

The  customers  with  whom  AIS  works  look  to  the  organization  to  develop  state-of-the-art 
systems.  The  majority  of  commercial  system  development  is  an  unprecedented  effort  as  engi¬ 
neers  utilize  cutting-edge  technology  in  their  development  work.  The  systems  are  from  many 
different  application  domains.  Platforms  vary  from  the  Internet  to  client-server  to  mainframe. 
Programming  languages  used  include  C,  C++,  COBOL,  Java,  and  Visual  BASIC.  The  devel¬ 
opment  projects  for  which  AIS  provides  resources  can  be  classified  as  either  custom  com¬ 
mercial  or  embedded  systems.  These  projects  are  typically  staffed  with  teams  ranging  in  size 
from  one  to  six  people. 

1.2  Business  Needs  and  Software  Process 
Improvement 

The  continuing  commitment  of  AIS  to  process  improvement  is  motivated  by  the  business 
needs  to: 

•  Deliver  defect-free  software  products  and  satisfy  customers’  increasingly  demanding 
time-to-market  goals. 

•  Achieve  a  sustained  competitive  advantage  in  the  consulting  business  by  becoming  a 
provider  of  teams  of  process  trained  engineers. 

•  Minimize  the  impact  of  staff  turnover. 

The  software  process  improvement  initiative  at  AIS  started  in  January  1992.  The  goals  were 
to: 


•  Improve  profitability  of  development  projects  by  meeting  cost  estimates  and  schedule 
commitments  with  reasonable  consistency. 
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•  Provide  a  continuing  management  focus  on  the  progress  and  visibility  of  each  project 
from  initial  commitment  to  orderly  progression  through  the  development  life-cycle 
phases  and  customer  acceptance. 

•  Enable  continuous  improvement  of  the  development  process  through  a  changed 
organizational  culture  biased  towards  rapid  implementation  of  many  small  incremental 
improvements  as  opposed  to  a  few  large  changes. 

1.3  Process  Improvement  Strategy 

AIS  named  its  process  improvement  initiative  Continuous  Process  Improvement  (CPI).  The 
company  chose  the  SEI  Capability  Maturity  Model  (CMM)1  as  the  process  maturity  frame¬ 
work  to  improve  organizational  process  capability  and  the  Institute  of  Electrical  and  Elec¬ 
tronics  Engineers  (IEEE)  standards  as  the  guidelines  for  software  engineering.  The  Personal 
Software  Process  (PSP)2  was  adopted  as  the  enabling  technology  to  improve  individual  engi¬ 
neer  performance  and  productivity.  Process  definition  gradually  evolved  and  matured  across 
maturity  levels  and  key  process  areas  (KPAs)  through  a  simple  and  effective  mechanism  of 
the  Process  Improvement  Proposal  (PIP).  To  evaluate  and  implement  the  PIPs,  a  Software 
Engineering  Process  Group  (SEPG),  utilizing  the  skills  and  experience  of  many  engineers, 
was  established  with  the  allocation  of  these  resources  being  on  a  part-time  basis. 

Additionally,  Watts  Humphrey’s  Managing  the  Software  Process  [Humphrey  89]  was  used  as 
a  guide.  The  approach  was  to: 

•  Conduct  self-assessments  with  a  focus  on  action. 

•  Utilize  the  skill  and  experience  of  many  people  to  implement  the  findings  and 
improvements. 

•  Utilize  a  PIP  process  to  enable  and  guide  the  gradual  evolution  of  AIS  processes. 

•  Maintain  organization  awareness  of  improvement  efforts  through  quarterly  status 
reviews. 


1  The  Capability  Maturity  Model  (CMM)  is  a  model  developed  through  the  efforts  of  the  SEI,  which 
provides  software  organizations  with  the  guidance  necessary  to  gain  control  of  their  processes  for 
developing  high-quality  software.  Based  on  the  software  process  maturity  framework,  the  CMM  is 
designed  to  assist  software  organizations  in  the  selection  of  process  improvement  activities  by 
determining  their  current  process  maturity. 

2  The  PSP  is  a  structured  and  disciplined  framework  that  helps  engineers  plan,  track,  and  control  their 
individual  work.  It  assists  software  engineers  in  estimating  the  size  of  the  work  product  and  effort 
needed  to  complete  the  task,  and  it  provides  a  framework  to  produce  a  high-quality  product.  The  PSP 
utilizes  disciplined  practices  such  as  personal  design  reviews  and  code  reviews  to  enable  software 
engineers  to  remove  defects  early  in  the  life  cycle  of  a  work  product. 
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2  Continuous  Process  Improvement 
Mechanisms 


2.1  Organization  Infrastructure 

The  unshaded  boxes  in  Figure  1  represent  the  entities  that  comprise  the  Development  Group. 
These  resources  may  also  have  some  additional  corporate-wide  responsibilities.  The  AIS 
President  provides  resources,  approvals,  and  policy  direction  to  the  Development  Manager. 
The  AIS  Development  Manager  provides  approvals  and  direction  to  the  Development  Group 
entities  and  to  the  software  development  process.  The  Development  Manager  also  reviews 
status  and  issues  with  the  President.  Computing  Support  and  Internet  Services  manage  the 
development  environment.  The  Project  Managers  and  software  engineers  enact  the  software 
development  process.  Project  Managers  review  plans,  issues,  and  status  with  the  Develop¬ 
ment  Manager.  The  Subcontract  Manager  is  responsible  for  administering  the  terms  of  the 
master  subcontract  agreement  with  the  AIS  subsidiary  in  Chennai,  India. 

The  Software  Quality  Assurance  (SQA)  Group  is  responsible  for  audit  and  review  of  the  de¬ 
velopment  process  and  for  sharing  its  findings  with  the  Development  Manager.  SQA  notifies 
the  President  of  issues.  The  SEPG  performs  Interim  ProfileSM  assessments  of  all  current  proj¬ 
ects.  The  group  also  reviews  PIPs  to  assess  what  impact  implementation  may  have  on  the 
organization’s  process.  If  the  determination  is  made  that  the  proposal  meets  the  company’s 
criteria  for  implementation,  work  is  done  to  make  it  a  part  of  the  company  process.  The 
Software  Configuration  Management  (SCM)  Group  provides  configuration  management  and 
change  control  support.  Project  Managers  and  software  engineers  staff  the  SQA,  SCM,  and 
SEPG  functions. 


SM  Interim  Profile  is  a  service  mark  of  Carnegie  Mellon  University. 
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Figure  1:  Organization  Infrastructure 

2.2  Project  Processes  and  Capability  Maturity  Model 

In  1992,  the  AIS  CPI-documented  processes  were  incorporated  into  a  one-volume  manual. 
This  was  organized  with  a  set  of  policies  in  the  introductory  section.  Management  and  soft¬ 
ware  engineering  processes  and  standards  were  added  in  sequential  order  as  they  were  devel¬ 
oped.  Version  1 .0  of  the  CPI  Manual  adopted  existing  IEEE  standards  for  life  cycle,  require¬ 
ments  specification,  quality  assurance,  configuration  management,  verification  and 
validation,  and  testing.  The  policies  regarding  those  processes  acted  as  pointers  to  the  IEEE 
standards  that  were  elsewhere  in  the  library. 

A  set  of  detailed  Work  Breakdown  Structures  (WBS  1.0),  an  integration  of  best  practices 
from  several  successful  projects,  were  added  during  this  time  period.  Estimating  spreadsheets 
that  mirrored  the  activities  in  the  WBS  were  developed  to  help  with  planning  size  and  effort 
for  each  project  phase.  (The  SEPG  collects  data  from  these  spreadsheets  to  update  the  stan¬ 
dard  productivity  rates  that  are  part  of  the  spreadsheets.)  New  processes  were  added  and  ex¬ 
isting  processes  were  modified  as  a  result  of  PIPs  and  self-assessment  findings. 

In  1995,  it  was  decided  that  process  improvement  efforts  would  be  more  effective  if  the  proc¬ 
esses  were  organized  in  the  same  manner  as  the  improvement  model  that  was  being  used.  The 
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CPI  was  restructured  by  dividing  the  process  manual  into  three  volumes  to  match  the  three 
process  categories  in  the  key  practices  of  the  CMM,  Version  1.1.  The  Management,  Organ¬ 
izational,  and  Engineering  Manuals  were  developed  by  organizing  each  volume  by  key  proc¬ 
ess  area  and  sequenced  by  maturity  level.  Each  AIS  process  was  then  mapped  to  a  key  proc¬ 
ess  area.  Processes  that  did  not  appear  to  pertain  to  a  KPA  were  analyzed  and  moved  to  other 
company  administrative  documentation.  The  processes  were  then  organized  with  all  relevant 
documentation  located  in  the  same  subject. 

By  1995,  over  50  PIPs  had  been  written  to  improve  the  WBS  and  it  had  evolved  to  Version 
1.4.  The  WBS  then  consisted  of  414  tasks.  Indications  were  that  an  analysis  should  be  con¬ 
ducted  to  determine  which  WBS  tasks  were  actually  being  used  by  projects.  The  unplanned 
activities  being  added  to  projects  were  also  analyzed.  This  information,  along  with  recent 
PIPs  and  feedback  from  Project  Managers,  was  used  to  remove  redundancies  and  to  consoli¬ 
date  similar  small  effort  tasks.  Modifications  encompassed  the  inclusion  of  small  effort  tasks 
into  checklists,  the  identification  of  tasks  that  should  be  pointers  to  scripts,  and  the  creation 
of  a  process  for  tailoring. 

As  a  result  of  the  introduction  of  PSP,  it  was  necessary  to  integrate  PSP  tasks  into  the  WBS 
and  associated  processes.  Project  Managers  presented  draft  iterations  of  WBS  2.0  with  the 
final  draft  consisting  of  207  tasks.  The  architecture  of  the  WBS  was  changed  to  be  consistent 
with  the  new  architecture  of  the  CPI  Manual.  The  project  management  tasks  were  moved  to 
one  template  separate  from  the  software  engineering  templates.  The  software  engineering 
templates  were  revised  for  each  life-cycle  phase.  The  project  management  template  consisted 
of  tasks  that  were  grouped  by  KPA.  Each  WBS  task  contained  a  reference  to  a  CMM  key 
process  area  goal  or  key  practice. 

The  Development  Group’s  standard  process  is  documented  in  hard  copy.  The  SEPG  is  the 
owner  of  the  process  documentation  and  is  responsible  for  updates.  New  versions  are  targeted 
for  release  to  coincide  with  Quarterly  Status  Reviews.  In  January  of  1999,  the  CPI  processes 
were  made  available  online  through  the  CPI  on  the  Web  (CPIW)  application. 

2.2.1  Project  Management 

Project  management  has  evolved  to  be  a  well-defined,  highly  integrated  set  of  processes  that 
are  used  throughout  the  life  cycle  of  a  project.  (See  Figure  2.) 
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Figure  2:  Project  Planning  Process  (1) 

The  Pre-Commitment  Process  is  executed  when  an  initial  written  or  verbal  Request  for 
Proposal  is  received.  This  process  is  also  initiated  prior  to  the  beginning  of  each  phase  of  a 
project.  It  is  the  mechanism  to  obtain  a  commitment  from  senior  management,  the  customer, 
development  team,  and  other  affected  groups  such  as  Software  Quality  Assurance  and  Soft¬ 
ware  Configuration  Management.  During  the  Pre-Commitment  Process,  the  Risk  Identify, 
Analyze,  and  Plan  phases  of  the  Risk  Management  Process  are  completed.  The  output  from 
the  process  (Risk  Identify,  Analyze,  and  Plan  form)  is  used  to  communicate  with  the  Senior 
Executive  and  to  gain  an  understanding  of  how  to  proceed  with  the  project.  Assumptions 
from  this  process  are  integrated  into  the  Statement  of  Work  Process,  and  mitigating  activities 
or  tasks  are  added  to  the  project  plan. 

The  Statement  of  Work  Process  is  executed  in  stages.  Figure  3  shows  a  relationship  be¬ 
tween  the  stages.  Determinations  as  to  the  actual  work  to  be  performed,  identification  of  the 
customer,  dates  for  completion,  and  the  criteria  for  determining  the  success  are  established 
during  the  Goals  and  Objectives  stage.  During  the  Work  Breakdown  Structure  stage,  the  team 
tailors  the  AIS  standard  WBS  to  fit  the  project  and  agrees  on  the  activities  and  tasks  to  be 
estimated.  Standard  estimating  spreadsheets  and  historical  data  are  used  to  estimate  the  size, 
effort,  schedule,  and  cost  for  the  activities  and  tasks  in  the  WBS.  Each  software  engineer  uses 
the  PSP  and  his  or  her  own  historical  data  to  estimate  the  size,  effort,  and  schedule  when 
there  are  components  to  be  built.  These  estimates  are  used  in  planning  with  some  adjustment 
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for  dependencies,  such  as  team  inspections.  The  Statement  of  Work  (SOW)  is  then  written 
(assembled)  and  reviewed  with  the  customer. 
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Figure  3:  Project  Planning  Process  (2) 

After  a  commitment  to  proceed  with  the  project  is  received  from  the  customer,  the  Project 
Plan  Process  is  executed.  This  process  involves  refining  the  plan  with  input  from  the  cus¬ 
tomer  review  of  the  SOW.  Project  startup  administrative  tasks  such  as  the  Time  Accounting 
Process,  Project  History  Book,  and  Computing  Support  requests  are  also  initiated. 

Figure  4  shows  how  the  Development  Tracking  Process  is  utilized  during  each  phase  as  the 
mechanism  for  project  tracking  and  oversight  and  for  determining  corrective  actions.  Each 
week  the  Development  Manager  meets  with  each  Project  Manager  to  assess  quality  by  re¬ 
viewing  the  defects  found  and  fixed  in  each  phase  of  the  process.  They  also  assess  the  Sched¬ 
ule  Plan  vs.  Actual  and  Effort  Plan  vs.  Actual  by  reviewing  the  earned  value  tracking  charts. 
Unplanned  activities  and  intergroup  coordination  activities  are  discussed.  The  risks  that  were 
identified  during  the  Risk  Identify,  Analyze,  Plan  Process  are  reviewed  on  a  periodic  and 
event-driven  basis,  and  new  mitigation  strategies  are  determined. 
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Software  Configuration  Management 


_ Software  Quality  Assurance _ 

_ Requirements  Management _ 

(CE  =  Concept  Exploration,  RE  =  Requirements  Engineering,  PD  =  Preliminary  Design,  DfC/T  =  Design/Code/Test, 
I/C=  InstaUatioiVCheckout,OS=  Operational  Support) _ 


Figure  4:  Project  Tracking  &  Control  Processes 

Software  Configuration  Management,  Software  Quality  Assurance,  and  Requirements  Man¬ 
agement  are  included  as  activities  in  the  WBS  for  each  life-cycle  phase.  At  the  Phase  Review, 
which  is  held  at  the  end  of  each  life-cycle  phase,  the  quality,  effort,  and  schedule  results  are 
presented  to  the  customer  along  with  the  final  deliverables  as  stated  in  the  Statement  of  Work. 
The  customer  is  given  a  Customer  Feedback  form  for  rating  and  commenting  on  the  project 
phase.  During  the  Quarterly  Status  Review  (QSR),  the  Project  Manager  from  each  project 
presents  quality,  effort,  and  schedule  status  to  the  Development  Group. 


2.2.2  Software  Engineering 

Software  Engineering  Work  Breakdown  Structures  exist  for  the  software  engineering  tasks  of 
Requirements  Engineering,  Preliminary  Design,  Design/Code/Test,  Installation/Checkout, 
and  Operational  Support  phases.  A  Work  Breakdown  Structure  for  objected-oriented  analysis 
and  design  has  been  piloted  in  projects,  and  will  be  incorporated  into  the  AIS  process  in  the 
future.  Deliverables  for  each  of  the  software  engineering  tasks  are  defined  along  with  proc¬ 
esses  and  standards.  All  phases  of  the  life  cycle  require  formal  inspections,  and  these  are  ref¬ 
erenced  in  the  Work  Breakdown  Structures.  Planning,  Overview,  Preparation,  Meeting,  Re¬ 
work,  and  follow-up  stages  organize  the  formal  inspection  process.  Guidelines,  scripts,  and 
forms  are  incorporated  into  the  process. 

AIS  software  engineers  use  PSP  during  the  Design/Code/Test  phase  of  our  software  life  cy¬ 
cle.  (See  Figure  5.)  Activities  include: 
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•  Engineers  estimate  their  own  modules  using  their  historical  data  on  size  estimates, 
productivity  rates,  and  effort  estimates. 

•  Engineers  create  a  plan  for  each  module. 

•  Engineers  submit  their  plans  to  the  Project  Manager. 

•  Project  Manager  produces  the  project  plan. 


Figure  5:  Software  Engineering  Processes 
During  the  development  of  the  product,  the  following  occur: 

•  Engineers  follow  PSP  life-cycle  phases:  Design,  Personal  Design  Review,  Team  Design 
Inspection,  Code,  Personal  Code  Review,  Team  Code  Inspection,  Compile,  and  Unit  Test. 

•  Personal  checklists  are  used  for  design  and  code  reviews. 

•  Defects  are  recorded  during  all  the  phases  of  the  work  product. 

•  Checklists  are  updated  based  on  defect  analysis. 
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•  Engineers  track  their  progress  using  Planned  Value  vs.  Earned  Value3  and  Planned  Effort 
vs.  Actual  Effort. 

•  Engineers  report  their  progress  to  the  Project  Manager  every  week. 

•  Project  Manager  updates  the  project  plan  and  reports  the  status  of  the  project  to  the 
Development  Manager. 

•  Upon  completion,  engineers  record  the  actual  size  of  the  work  product,  and  actual  time 
taken  to  complete  the  product,  and  analyze  size,  effort,  and  defect  data  to  make  necessary 
improvements.  (See  Figure  6  and  Figure  7.) 

•  Subsequent  to  engineers  reporting  their  data,  the  Project  Manager  compiles  all  the 
information  to  reflect  Planned  vs.  Actual  Earned  Value  and  Planned  vs.  Actual  Hours 
from  a  total  project  perspective  as  shown  in  Figure  8  and  Figure  9. 
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Figure  6:  Individual  Planned  vs.  Actual  Earned  Value 


3  Earned  Value  tracking  is  a  method  for  evaluating  a  project’s  progress  by  establishing  relative  value 
for  each  task  to  be  completed.  By  using  the  Earned  Value  system,  a  common  measure  can  be  used  to 
judge  the  progress  of  a  project.  A  common  value  scale  is  assigned  to  each  task,  regardless  of  the  type 
of  work.  Total  hours  for  the  entire  project  are  estimated,  and  each  task  is  assigned  an  earned  value 
based  on  its  individual  estimated  percentage  of  the  total  hours.  Earned  Value  is  credited  only  when  a 
task  is  fully  completed.  If  a  task  is  too  big,  it  can  be  broken  down  into  subtasks.  By  doing  this,  the 
project  will  be  able  to  show  some  Earned  Value  for  the  completion  of  the  subtask. 
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Figure  9:  Project  Planned  vs.  Actual  Hours 

2.3  Organization  Processes 

2.3.1  Process  Improvement  Proposal 

The  PIP  process  complements  the  evolutionary  process  maturity  framework  inherent  in  the 
CMM  by  ensuring  that  each  one  of  many  small  changes  introduced  is  based  on  practitioners’ 
experience.  In  July  1992,  AIS  adopted  PIP  forms  in  order  to  empower  engineers  to  evolve 
AIS  processes  by  means  of  suggesting  small  incremental  improvements.  The  PIP  form  pro¬ 
vides  a  direct  correlation  between  the  CMM  KPA  and  the  improvement  benefit  to  the  AIS 
process.  Where  possible,  these  benefits  are  quantified.  The  SEPG  was  made  responsible  for 
evaluating  PIPs,  identifying  resources  to  implement  improvements,  providing  feedback  to  the 
authors,  maintaining  PIP  records,  and  reporting  PIP  status  in  the  QSR. 

2.3.2  Training  Program 

The  commitment  to  required  software  engineering  training  institutionalizes  the  software  pro¬ 
cess  to  existing  staff,  as  well  as  newly  hired  staff.  It  is  the  key  mechanism  that  enables  AIS  to 
supply  the  industry  with  teams  of  process-trained  engineers.  The  AIS  Training  Program  con¬ 
sists  of  three  tracks:  Core  Processes,  Job  Requirements,  and  Career  Path.  The  training  in  the 
job-related  and  career  path  tracks  is  delivered  to  meet  project  needs  or  an  individual’s  career 
path  goals.  These  courses  are  generally  outsourced.  AIS  staff  teaches  classes  in  object- 
oriented  analysis  and  design  and  Java.  The  following  software  engineering  courses  are  re¬ 
quired  for  all  positions  related  to  software  development  and  are  taught  at  AIS: 

•  Managing  the  AIS  Software  Process  -  Introduces  software  quality  issues  and  the 

processes  necessary  for  continuous  improvement.  Software  process  maturity  is  defined 
within  the  framework  of  the  SEI  CMM.  AIS  processes  and  functions  such  as  SQA,  SCM, 
and  SEPG  are  introduced  in  relationship  to  the  CMM  key  process  areas.  Assignments 
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include  creation  of  a  project  plan.  Text:  Managing  the  Software  Process  by  Watts 
Humphrey  [Humphrey  89] 

•  Personal  Software  Process  -  Defines  the  software  engineering  process  in  a  series  of 
seven  personal  processes  beginning  with  a  basic  process  (PSPO)  and  building  to  a  process 
capable  of  supporting  larger  scale  development  (PSP3).  The  engineers  learn  to  plan, 
track,  and  improve  their  own  processes  and  how  to  improve  quality  by  using  defect 
management.  Assignments  consist  of  8  to  10  programming  assignments  and  5  report 
exercises.  Text:  A  Discipline  for  Software  Engineering  by  Watts  Humphrey  [Humphrey 
95] 

•  Requirements  Engineering  -  Defines  requirements  engineering  as  the  process  of 
elicitation,  analysis,  specification,  and  verification,  and  as  the  product — the  Software 
Requirements  Specification  based  on  IEEE  standards.  Methods,  tools,  and  issues  are 
discussed.  Resources:  Industry  expert  and  AIS  workshop  materials 

•  Inspection  Process  -  Explains  the  concepts  and  principles  behind  software  inspections, 
introduces  the  AIS  inspection  process,  and  provides  practice  with  AIS  inspection  process. 
Text:  Software  Inspection  Process  [Ebenau  94] 

2.4  Process  Maturity  Assessment 

2.4.1  Self-Assessment 

A  formal  assessment  did  not  seem  financially  feasible  at  the  onset  of  the  company’s  process 
improvement  efforts.  In  lieu  of  a  formal  assessment,  the  decision  was  made  to  utilize  the  SEI- 
technical  report,  A  Method  for  Assessing  the  Software  Engineering  Capability  of  Contractors 
[Humphrey  87],  All  members  of  the  Development  Group  completed  the  questionnaire  associ¬ 
ated  with  this  report.  Project  teams  were  interviewed  to  identify  and  prioritize  the  major  as¬ 
sessment  findings.  At  the  conclusion  of  these  efforts,  the  determination  was  made  that  the 
organization  was  operating  at  the  Ad  Hoc  level  as  an  organization.  The  major  findings  were 
mapped  to  Humphrey’s  suggestions  for  an  organization  to  improve  its  processes  with  the  ob¬ 
jective  of  moving  from  the  Ad  Hoc  level  to  the  SEI CMM  Level  2  key  process  areas.  The 
Project  Management  System  was  the  focus  for  advancing  from  the  Ad  Hoc  level.  The  areas 
identified  as  target  items  on  which  to  focus  were: 

•  Commitment  Review 

•  Quarterly  Status  Review 

•  Phase  Review 

The  organization  elected  to  have  resources  expend  effort  in  three  KPAs  from  CMM  Level 
2 — Project  Planning,  Project  Tracking,  and  Software  Configuration  Management.  The  proc¬ 
ess  areas  were: 

•  Project  Process 

•  Project  Tracking  Process 

•  Environment  Documentation 
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The  findings  were  followed  up  with  specific  aforementioned  actions  of: 

•  Establishment  of  the  CPI  Initiative 

•  Creation  of  the  CPI  Manual 

•  Organization  of  the  Software  Engineering  Process  Group  (SEPG) 

The  first  QSR  was  held  in  March  1992.  The  same  model  was  used  for  improvement  through 
1993,  using  the  questionnaire  to  provide  internal  process  assessments  every  six  months,  to 
report  on  the  results,  and  to  plan  improvements  from  the  findings. 

In  1994,  the  SEI  Interim  Profile  process  was  tailored  for  use  by  AIS.  The  current  process  is 
for  individual  project  team  members  to  complete  the  Software  Project  Maturity  Questionnaire 
semiannually.  From  this  questionnaire,  the  SEPG  constructs  a  draft  project  profile,  meets 
with  the  team  to  review  results,  and  works  to  resolve  discrepancies.  A  final  project  profile  is 
then  prepared.  The  final  project  profiles  are  rolled  up  into  the  organization  profile.  Project 
Managers  present  project  profiles  in  the  QSR,  and  the  SEPG  presents  the  organization  pro¬ 
file. 

2.4.2  CMM-Based  Assessment  for  Internal  Process 
Improvement 

In  April  1996,  AIS  participated  in  a  CMM-Based  Assessment  for  Internal  Process  Improve¬ 
ment  (CBA IPI)  with  an  external,  independent  SEI-licensed  lead  assessor.  The  objectives  of 
the  appraisal  were  to  identify  the  strengths  and  weaknesses  of  the  AIS  software  process,  to 
identify  the  highest  priority  issues  for  software  process  improvement,  and  to  provide  a 
framework  for  process  improvement  actions.  The  scope  was  to  assess  five  projects  where  AIS 
had  primary  responsibility  for  project  and  process  management  and  to  evaluate  the  AIS  soft¬ 
ware  process  in  all  13  of  the  key  process  areas  of  the  CMM  for  Repeatable  and  Defined  lev¬ 
els. 

The  following  global  strengths  were  noted  during  the  assessment: 

•  Processes  at  AIS  have  undergone  evaluation  and  have  evolved  since  1992.  There  is 
strong  evidence  of  the  correlation  between  project  profitability  and  process  fidelity.  With 
the  organizational  culture  and  process  in  place,  it  is  expected  that  the  benefits  of  process 
improvement  effort  will  continue  into  the  future. 

•  Process  improvement  efforts  are  included  in  performance  evaluations  (first  item  in 
position  accountability  statements). 

•  The  SQA  function  has  gained  the  confidence  of  the  projects  as  a  value-added  role. 

•  Software  engineers  use  the  PSP  in  most  aspects  of  their  jobs. 

Improvement  opportunities  were  identified  in  the  KPAs  of  Software  Configuration  Manage¬ 
ment,  Software  Quality  Assurance,  Subcontract  Management,  and  Organization  Process 
Definition.  The  remainder  of  the  KPAs  for  Levels  2  and  3  were  fully  satisfied.  The  PSP  ad- 
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dresses  a  number  of  these  satisfied  KPAs  including  Software  Project  Planning  and  Software 
Project  Tracking  and  Oversight  at  Level  2,  as  well  as  Organization  Process  Focus,  Integrated 
Software  Management,  Software  Product  Engineering,  and  Peer  Reviews  from  Level  3. 

There  seems  to  be  a  correlation  between  the  KPAs  that  PSP  addresses  and  the  satisfied  KPAs. 

An  action  plan  based  on  the  improvement  opportunities  identified  in  the  assessment  was  pre¬ 
sented  to  the  Development  Group  at  the  June  1996  QSR.  The  action  plan  identified  each  task, 
planned  schedule  for  implementation,  and  resources  assigned.  In  addition  to  improvement 
opportunities  identified  at  the  Repeatable  and  Defined  levels,  the  action  plan  contained  tasks 
related  to  moving  the  organization  to  the  Managed  (Quantitative)  level.  The  status  of  the  plan 
was  reviewed  at  the  beginning  of  every  subsequent  QSR  until  all  tasks  were  implemented. 
Adjustments  in  resources  were  made  to  keep  the  plan  on  schedule.  The  Interim  Profile  proc¬ 
ess  is  to  be  conducted  semiannually.  A  reassessment  (CBA IPI)  was  planned  for  April  1999. 

The  follow-up  assessment  was  conducted  in  April  1999.  The  primary  purpose  of  the  appraisal 
was  to  provide  recommendations  on  which  to  focus  AIS  future  process  improvement  actions. 
The  primary  goal  of  the  assessment  was  not  to  provide  a  CMM  level  rating,  although  one  of 
the  objectives  was  to  ascertain  this  information.  The  scope  again  was  limited  to  AIS  projects 
where  AIS  had  the  primary  responsibility  for  project  and  process  management.  Four  current 
projects  were  included  in  the  assessment  and  were  evaluated  in  all  15  of  the  KPAs  for  the 
CMM  levels  Repeatable,  Defined,  and  Managed  levels  as  well  as  the  Defect  Prevention  KPA 
from  the  Optimizing  Level. 

As  a  result  of  the  assessment,  the  following  global  strengths  were  identified: 

•  Quality  and  process  culture  supports  personnel  versatility  and  movement  within  the 
organization. 

•  Process  culture  also  provides  organizational  stability  while  moving  into  new  application 
domains. 

•  Training  infrastructure  provides  effective  knowledge  and  skills  necessary  for  personal 
growth  and  organizational  success. 

•  People  “make”  the  process. 

Evidence  gathered  during  the  assessment  verified  that  all  of  the  KPAs  for  Levels  2  and  3 
were  fully  satisfied.  Improvement  opportunities  were  identified  for  the  following  KPAs: 
Quantitative  Process  Management,  Software  Quality  Management,  and  Defect  Prevention. 
The  findings  from  the  assessment  determined  that  the  AIS  Development  Group  had  imple¬ 
mented  the  CMM  key  practices  and  had  satisfied  the  goals  found  in  Level  2  and  Level  3  of 
the  CMM.  An  action  plan  based  on  the  improvement  opportunities  is  being  formulated.  The 
action  plan  will  identify  an  activity,  resource(s),  and  schedule  for  each  improvement  opportu¬ 
nity  or  recommendation  that  was  an  outcome  of  the  assessment. 
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3  Results 


The  changes  in  AIS  process  capability  can  be  delineated  into  three  different  eras.  The  first  era 
was  the  time  period  before  1992,  at  which  time  AIS  did  not  use  a  model  for  software  process 
improvement.  In  the  second  era,  from  1992  to  1995,  the  company  implemented  the  CMM. 
Data  indicate  that  project  schedules  and  effort  predictability  improved  with  the  introduction 
of  the  CMM  maturity  framework.  The  third  era  began  in  1995  when  AIS  piloted  the  PSP.  The 
PSP  was  integrated  into  the  defined  process  in  1996.  Data  gathered  reflected  an  increase  in 
product  quality,  as  well  as  improvements  in  productivity  levels.  The  consequent  reduction  in 
rework  led  to  lower  life-cycle  cost  and  greater  profitability. 

The  data  shown  are  for  commercial  systems  that  AIS  has  been  developing  since  1986.  The 
data  for  the  embedded  system  development  are  not  included  with  the  commercial  system  data 
because  the  application  domain  is  quite  different.  The  embedded  system  work  was  a  series  of 
components  each  developed  by  an  individual  through  the  entire  life  cycle.  Each  component 
was  delivered  to  the  customer  as  a  separate  entity.  The  results  for  embedded  systems  will  be 
discussed  in  the  PSP  section. 

3.1  Schedule  and  Effort  Predictability 

Figure  10  graphically  depicts  the  impact  of  the  CPI  initiative  and  activities  on  projects’ 
schedule  commitments.  Before  1992,  successful  completion  of  projects  was  dependent  upon 
the  individual  ability  and  Herculean  efforts  of  a  few  Project  Managers  and  engineers. 

The  AIS  Development  Group  has  progressed  from  an  ad  hoc  environment  with  unpredictable 
and  frequent  over-budget  efforts  to  an  environment  of  disciplined  commitments.  The  process 
limits  in  Figure  10  show  the  change  in  capability  to  predict  schedule  from  1988  through 
1998.  Although  the  company  was  established  in  1986,  data  were  not  available  for  projects 
started  earlier  than  July  1988.  The  process  limits  were  calculated  using  a  moving  range.  The 
date  when  the  project  development  phase  started  was  used  as  the  horizontal  coordinate  be¬ 
cause  experience  indicates  that  the  process  maturity  of  the  project  phase  is  influenced  by  the 
maturity  of  the  organization  when  the  phase  started.  Each  data  point  represents  a  new  devel¬ 
opment  phase. 
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Schedule  Deviation  Individual  Value  Control  Chart  - 
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Figure  10:  Schedule  Deviation  Individual  Value  Control  Chart  -  Commercial 
Systems 

After  1992,  when  use  of  the  CMM  as  the  improvement  model  was  instituted,  schedules  be¬ 
came  more  predictable.  At  that  time,  project  schedules  were  based  on  the  AIS  defined  pre¬ 
commitment  process  with  formal  executive  review  and  involvement  of  Project  Managers  and 
engineers  in  project  planning.  CPI  activities  such  as  Quarterly  Status  Review,  Phase  Review, 
and  Software  Quality  Assurance  ensured  project  visibility.  Executive  oversight  of  the  devel¬ 
opment  process  provided  a  significant  positive  impact  on  the  project’s  ability  to  meet  sched¬ 
ule  commitments. 

The  data  indicate  that  the  average  schedule  deviation  improved  from  112%  in  the  pre-model 
era  to  41%  in  the  CMM  era  to  5%  in  the  CMM+PSP  era.  Schedule  predictability  again  im¬ 
proved  when  each  software  engineer  used  PSP  to  plan  the  schedule  for  his  or  her  component 
and  made  that  commitment  to  the  Project  Manager  and  overall  project  plan. 

Figure  1 1  graphically  depicts  the  impact  of  the  CPI  initiative  and  activities  on  a  project’s  ef¬ 
fort  commitments.  While  meeting  the  schedule  met  the  customer’s  needs,  much  of  the  work 
was  contracted  on  a  fixed-cost  bid  and,  if  effort  was  not  predictable,  the  project  phase  was 
not  profitable. 

When  AIS  began  using  the  PSP,  in  addition  to  continuing  CMM-based  improvement,  effort 
became  more  predictable.  Some  of  the  effort  predictability  was  the  result  of  the  PSP-trained 
software  engineers  planning  their  own  work  and  utilizing  their  own  historical  data  to  esti¬ 
mate;  however,  the  most  significant  improvement  in  effort  predictability  occurred  because  of 
the  quality  practices  the  PSP-trained  software  engineers  applied. 
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The  data  reflect  the  average  effort  deviation  improved  from  87%  in  the  pre-model  era  to  37% 
in  the  CMM  era  to  -4%  in  the  CMM  +  PSP  era. 
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Figure  11:  Effort  Deviation  Individual  Value  Control  Chart  -  Commercial  Systems 

3.2  Quality  &  Productivity 

In  June  1995,  the  PSP  was  piloted  on  a  project  and  was  later  incorporated  into  the  AIS  de¬ 
fined  process.  Figure  12  shows  that  projects  using  the  PSP  have  significant  reductions  in 
System  Test  duration.  PSP-trained  engineers  have  few  or  no  acceptance  test  and  usage  de¬ 
fects.  Similar  results  were  not  achieved  prior  to  PSP  adoption.  The  consequent  near  elimina¬ 
tion  of  rework  time  and  effort  flows  directly  to  the  company  bottom  line. 

Since  adopting  the  PSP,  acceptance  tests  that  are  executed  by  the  customer  have  also  been  of 
short  duration  because  of  little  or  no  rework.  On  the  pilot  project  where  AIS  introduced  the 
PSP  in  1995,  some  of  the  software  engineers  had  been  trained  in  PSP  and  some  had  not.  The 
experience  and  education  of  the  engineers  in  both  groups  were  similar.  Acceptance  test  de¬ 
fects  were  tracked  to  the  components  developed  by  each  engineer,  and  results  showed  that  the 
PSP-trained  software  engineers  produced  a  higher  quality  product.  Figure  13  reflects  these 
data. 
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Figure  12:  System  Test  Data 


Figure  13:  Acceptance  Test  Defects:  PSP  vs.  Non-PSP 

Figure  14  illustrates  how  the  use  of  a  disciplined  process  enabled  the  PSP-trained  engineers 
to  be  more  productive.  The  time  used  for  the  LOC/hour  calculation  is  the  total  time  for  the 
production  of  components.  For  each  component,  the  time  was  tracked  for  all  phases  of  that 
component  including  design,  code,  and  test  activities  and  all  planning,  reviews,  and  inspec¬ 
tions. 
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Figure  14:  Productivity:  PSP  vs.  Non-PSP 

Figure  15  contains  defect  removal  data  from  projects  that  have  completed  the  entire  devel¬ 
opment  life  cycle  since  the  integration  of  tracking  defects  by  phase  into  the  process.  Individ¬ 
ual  PSP  reviews  are  used  along  with  formal  inspections  throughout  the  life  cycle  of  a  product 
and  have  proven  to  be  an  effective  defect-containment  strategy. 

3.3  Individual  Software  Engineer 

When  AIS  ventured  into  embedded  software  domain,  the  process  framework  and  PSP  as¬ 
sisted  in  improving  development  practices.  Figure  16,  Figure  17,  Figure  18,  and  Figure  19 
show  the  data  on  embedded  software  development  done  by  an  individual  software  engineer. 
This  was  an  experienced  individual  who  had  knowledge  of  PSP;  therefore,  there  was  a  higher 
degree  of  quality  in  the  work  from  the  beginning  of  development.  These  charts  clearly  dem¬ 
onstrate  the  improvement  in  estimating  accuracy  of  size  and  effort.  Unit/component  test  de¬ 
fects  were  around  one  defect/KLOC.  During  the  development  of  these  components,  the  engi¬ 
neer  showed  a  slight  gain  in  productivity  because  of  no  rework  tail. 
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Defects  Removed  By  Phase 


Figure  15:  Defects  Removed  by  Phase 


Figure  16:  Size  Estimating 
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Figure  18:  Productivity 
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Figure  19:  Test  Defects 

3.4  Process  Culture 

Continuous  improvement  of  the  development  process  occurs  because  of  an  organizational 
culture  that  understands  continuous  improvement  and  its  relationship  to  the  CMM,  and  par¬ 
ticipates  in  proposing  and  implementing  small  incremental  improvements.  Figure  20  shows 
the  number  of  PIPs  implemented  by  CMM  key  process  area  since  1992.  Of  the  current  group, 
95%  of  all  engineers  who  have  been  with  the  group  more  than  6  months  have  submitted  at 
least  one  PIP. 
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From  the  period  July  1992  to  October  1998,  engineers  submitted  635  individual  PIPs.  The 
SEPG  implemented  412  PIPs  in  an  orderly  and  institutionalized  manner.  Sixty-three  individ¬ 
ual  engineers  have  submitted  PIPs  indicating  that  CPI  is  broad  based  within  the  AIS  Devel- 
opment  Group. 

Data  also  provide  evidence  that  participation  in  process  improvement  efforts  benefits  indi¬ 
vidual  career  enhancement  prospects.  Engineers  submitting  the  most  PIPs  have  been  pro¬ 
moted  to  President,  Development  Manager,  Process  Manager,  and  Training  Director. 

The  impact  of  CPI  has  been  most  evidenced  in  the  change  of  attitude  within  the  organization. 
It  is  no  longer  necessary  to  explain  the  value  of  process  improvement  to  employees — it  is  a 
way  of  life.  Evidence  of  this  is  found  in  the  comments  from  members  of  the  Development 
Group  in  response  to  the  question  “What  convinced  you  that  process  improvement  works?” 

3.4.1  Individual  Anecdotes 

One  software  engineer,  who  has  been  with  AIS  about  six  months,  described  her  experience  as 
compared  to  previous  employment.  “About  two  years  back  I  was  hired  for  incorporating 
changes  in  the  existing  information  system  of  a  major  life  insurance  company.  The  system 
was  so  huge  that  none  of  us  (including  the  on-site  managers)  had  an  in-depth  understanding 
of  the  entire  system  as  there  was  no  written  Software  Requirements  Specification.  We  were 
told  what  modules  to  make  the  changes  in.  But  it  would  take  hours  to  make  a  single  change 
because  we  had  to  look  through  undocumented  programs,  each  of  them  having  between  13 
KLOC  - 15  KLOC.  The  thought  “how  I  wish  the  code  was  documented”  crossed  my  mind 
not  just  once,  but  every  single  time  I  started  to  make  a  change!  And  we  were  aware  that  these 
changes  probably  had  ripple  effects  in  other  modules  that  needed  to  be  taken  care  of.  But  we 
had  no  clue  as  to  which  modules  were  getting  affected,  and  how  they  were  getting  affected. 
Moreover  fresh  requirements  were  being  gathered  on-site  as  we  were  busy  enhancing  the 
modules.  The  result  -  chaos!” 

She  continued,  “When  I  browsed  through  the  AIS  web  site  and  read  that  they  gave  high  pri¬ 
ority  to  quality  and  software  processes,  I  knew  where  I  had  to  go!  I  know  now  that  I  made  a 
right  decision  because  at  AIS  every  task  is  done  in  a  structured  and  organized  manner  using 
processes.  What  I  have  learned  from  my  experience  is  that  processes  are  not  only  important 
in  the  work  environment,  but  in  every  walk  of  life!” 

Another  software  engineer  who  has  been  with  us  for  several  years  told  her  story.  “I  was 
working  on  a  large  project  at  a  Fortune- 100  customer  when  our  customer  supervisor  came 
hurriedly  to  my  desk  and  said  that  she  wanted  some  information  about  our  project  urgently. 
At  that  time  my  AIS  Project  Manager  was  not  present,  but  my  team  member  and  I  could  pro¬ 
vide  all  the  information  she  wanted  about  the  project. 

The  customer  supervisor  wanted  to  know:  Is  it  on  schedule?  How  much  effort  was  spent? 
What  was  the  cost  at  this  point?  How  many  hours  were  spent  on  enhancements?  How  much 
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did  we  spend  on  Software  Change  Requests?  We  gave  her  the  information....  She  was  thor¬ 
oughly  impressed! 

This  incident  convinced  me  that  our  process  of  planning  and  tracking  projects  is  very  useful, 
not  only  to  us  but  to  our  customers  also!  This  made  me  realize  the  importance  of  process.” 

A  new  Project  Manager  shared  her  account  of  what  convinced  her  that  process  improvement 
works.  “In  recently  assuming  the  role  of  Project  Manager  of  a  new  development  project,  with 
little  experience  managing  a  project  using  software  processes  from  the  beginning  of  a  project 
to  its  close,  I  have  been  able  to  make  the  transition  in  responsibilities  with  a  relatively  small 
amount  of  effort.  I  firmly  believe  the  process  training  I  have  received  throughout  my  career 
here  at  AIS,  along  with  the  highly  qualified  process-trained  staff,  which  have  not  only  been 
my  supervisors  but  my  mentors,  are  the  two  major  contributing  factors  for  this  smooth  tran¬ 
sition. 

Although  I  have  worked  with  the  process  information  contained  in  the  Continuous  Process 
Improvement  on  the  Web  (CPIW)  application  in  my  role  as  a  Technical  Writer  for  the  com¬ 
pany,  I  see  it  as  a  whole  new  tool  from  this  role.  The  process  information,  templates,  forms, 
etc.,  contained  within  the  AIS  CPIW  have  not  only  provided  a  wealth  of  information  to  me, 
but  I  can  truly  see  how  these  documented  processes,  standards,  guidelines,  and  policies,  when 
used  properly,  can  result  in  significant  cost  savings  to  the  company  and  their  customers.” 

A  manager  at  AIS  who  started  as  a  software  engineer  in  1990  shared  his  experience.  “During 
my  first  two  years  at  AIS,  there  had  been  many  announcements  in  staff  meetings  about  how 
we  were  going  to  be  more  competitive,  deliver  quality  products,  and  make  deliveries  on  time 
(through  CASE  tools,  C++,  00,  etc.).  So  far  I  had  participated  on  two  projects  that  had  dis¬ 
appointed  key  customers  by  their  lateness  and  problems. 

In  January  1992,  our  AIS  Chief  Executive  Officer  (CEO)  went  to  a  conference  on  software 
process  improvement.  When  he  returned,  he  immediately  called  a  staff  meeting,  and  gave 
each  employee  a  copy  of  Managing  the  Software  Process.  We  were  all  asked  to  read  the  first 
six  chapters.  Our  “Continuous  Process  Improvement”  initiative  was  bom  and  became  the 
foundation  for  all  other  software  development  initiatives  at  AIS.  Using  the  principles  of  proc¬ 
ess  improvement,  we  did  a  better  job  -  not  by  trying  harder  or  working  faster — but  by  con¬ 
tinuously  learning  from  our  past  experiences  and  applying  this  to  future  work.” 

Other  comments  from  software  engineers  include:  “Everything  is  under  control  if  we  follow 
the  process.”  “Software  process  improvement  helped  me  realize  that  I  had  to  spend  more  time 
in  design,  and  thus  I  reduced  the  time  spent  in  test.” 
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4  Corporate  Links 


4.1  Balanced  Scorecard 

In  September  1995,  a  senior  management  group  was  organized  to  establish  a  corporate  long¬ 
term  strategy  based  on  surveys  of  customers,  employees,  and  shareholders.  The  Balanced 
Scorecard  [Kaplan  96]  framework  was  chosen  to  identify  the  core  outcome  measures  and 
performance  drivers  of  strategic  objectives  from  each  of  five  perspectives:  Financial,  Cus¬ 
tomer,  Employee,  Internal  Business  Process,  and  Learning  and  Growth.  The  objectives  and 
measures  now  are  part  of  a  management  system  that  links  them  to  each  other,  process  im¬ 
provements,  job  position  accountabilities,  and  business  strategy.  (See  Appendix  A:  AIS  Bal¬ 
anced  Scorecard.) 


The  specific  questions  for  which  answers  were  sought  in  order  to  formulate  a  company  strat¬ 
egy  are  listed  in  the  Balanced  Scorecard  Questions  shown  below. 


Perspective 

Question 

Financial 

What  is  important  to  our  shareholders? 

Customer 

What  is  important  to  our  customers? 

Employee 

What  is  important  to  our  employees? 

Internal  Business  Process 

To  satisfy  our  shareholders,  customers,  and  employees,  what 
business  processes  must  we  excel  at? 

Learning  &  Growth 

How  will  we  sustain  our  ability  to  change  and  improve? 

AIS  wanted  to  obtain  input  from  its  customers  to  understand  what  they  expected  in  a  soft¬ 
ware  product.  The  process  of  surveying  and  interviewing  its  customers  was  initiated.  Cus¬ 
tomer  feedback  indicated  that  they  were  expecting  defect-free  and  on-time  deliveries  of  soft¬ 
ware.  They  also  expected  to  get  value  for  their  money  and  expected  the  AIS  organization  to 
have  a  software  development  cycle  that  was  responsive  to  their  business  needs. 

Information  from  the  employee  perspective  was  obtained  by  surveying  employees  and  asking 
each  person  to  prioritize  the  key  process  areas  of  the  People  Capability  Maturity  Model 
(P-CMM)  [Curtis  95].4  The  surveys  were  compiled  and  the  six  key  process  areas  were  chosen 
with  five  of  the  six  being  from  the  Repeatable  level.  The  financial  perspective  was  written 

4  The  People  Capability  Maturity  Model  (P-CMM)  is  a  maturity  framework  developed  by  the  SEI  and 
patterned  after  the  CMM.  This  model  is  utilized  by  software  organizations  as  a  guide  for  continuously 
improving  the  management  and  development  of  their  human  assets.  The  P-CMM  provides  software 
organizations  with  the  insight  necessary  to  attract,  develop,  motivate,  organize,  and  retain  the  high- 
quality  personnel  necessary  to  improve  their  software  development  capability. 
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with  input  from  the  shareholders.  Consideration  was  given  to  feedback  from  the  shareholders, 
customers,  and  employees  in  order  to  define  the  internal  business  process  objectives.  The 
internal  business  process  perspective  is  based  on  CPI  of  the  projects,  individuals,  and  organi¬ 
zation.  The  support  for  these  improvements  comes  from  the  CMM,  the  PSP,  and  the  AIS  PIP 
process.  These  were  identified  as  the  means  by  which  critical  value  propositions  identified  by 
the  customers  would  be  delivered,  long-range  competitive  capabilities  would  be  built,  and 
superior  long-term  financial  performance  would  be  ensured.  The  internal  business  process 
results  were  discussed  in  Section  3. 

The  remaining  perspective  was  Learning  and  Growth.  To  satisfy  this  perspective,  the  com¬ 
pany  made  the  decision  to  invest  in  its  people,  process,  and  technology.  This  commitment 
was  driven  by  the  need  to  remain  innovative  and  to  satisfy  the  company’s  shareholders,  cus¬ 
tomers,  and  employees. 

Once  the  objectives  were  in  place,  a  set  of  measurements  was  defined — core  outcomes  and 
performance  drivers.  Performance  drivers  are  leading  indicators  that  are  measurable  early  in 
the  process,  while  core  outcomes  are  lagging  indicators.  For  instance,  for  the  learning  and 
growth  perspective,  “Engineers  use  the  PSP”  is  a  performance  driver,  and  “Engineers  im¬ 
prove  their  productivity  continuously”  is  a  core  outcome. 

After  the  core  outcomes  and  performance  drivers  were  completed,  a  target  was  defined  for 
each,  a  timeframe  for  taking  the  measurement  was  determined,  and  a  schedule  for  reporting 
the  results  was  established. 

4.1 .1  Financial  Perspective 

The  financial  strategic  objective  from  the  shareholders’  perspective  is  to  “Consistently  meet 
or  exceed  shareholder  expectations  for  revenue  growth,  profitability,  and  return  on  invest¬ 
ment.”  Individual  project  profitability  and  overall  AIS  revenue  and  profits  improved  signifi¬ 
cantly  as  our  process  improvement  progressed.  In  the  era  before  1992,  AIS  was  profitable  one 
out  of  the  five  years.  After  beginning  our  process  improvement  initiative  using  the  CMM, 
profit  margins  were  consistent.  During  the  years  between  1992  and  1995,  profit  as  a  percent¬ 
age  of  revenue  averaged  5.7%.  More  recently  in  the  CMM+PSP  era,  profit  as  a  percentage  of 
revenue  has  averaged  9.9%.  These  gains  have  occurred  at  the  same  time  that  the  AIS  organi¬ 
zation  was  experiencing  growth  from  21  people  in  1990  to  over  140  people  by  the  end  of 
1998. 

4.1 .2  Customer  Perspective 

AIS  has  seen  sustained  continuous  improvement  in  customer  satisfaction.  The  Balanced 
Scorecard  customer  strategic  objective  is  to  “Consistently  meet  or  exceed  customer  expecta¬ 
tions  for: 
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•  Defect-free  and  on-time  delivery  (quality) 

•  Value  for  products  and  services 

•  Achieving  time-to-market  goals  (timeliness) 

This  objective  is  measured  at  the  end  of  each  project  phase  by  using  a  Customer  Feedback 
Form  which  asks  the  customer  to  check  a  response  to  "AIS  has  delivered  value  products  and 
services  for  the  price  during  this  project  phase”  and  also  has  room  for  comments.  The  form 
that  was  used  in  1997-1998  had  choices  of  Achieved,  Need  to  Improve,  or  Did  Not  Achieve. 
Some  of  the  feedback  being  received  could  be  interpreted  as  a  rating  of  “Exceeded”;  how¬ 
ever,  responses  were  not  measured  with  the  feedback  form.  The  new  feedback  form  that  was 
adopted  as  a  response  to  a  PIP  in  1999  offers  choices  of  Exceeded  Expectations,  Met  Ex¬ 
pectations,  or  Did  Not  Meet  Expectations.”  Each  choice  is  provided  for  quality,  value,  and 
timeliness  measures. 

The  results  of  customer  feedback  forms  for  project  phases  in  1997-1998  were  as  follows: 

•  44  -  Achieved 

•  4  -  Need  to  Improve 

•  0  -  Did  Not  Achieve 

With  no  formal  marketing  organization,  another  indication  of  customer  satisfaction  is  that 
AIS  has  continued  to  experience  growth  through  repeat  business  and  industry  reputation. 

4.1.3  Employee  Perspective 

The  Balanced  Scorecard  employee  strategic  objective  is  to  “Consistently  meet  or  exceed  em¬ 
ployee  expectations  for  training,  compensation,  communication,  work  environment,  perform¬ 
ance  management,  and  career  development.” 

The  major  initiative,  “For  Your  Improvement”  (FYI)  started  in  October  1995,  to  improve  the 
maturity  of  people  practices  using  the  People  Capability  Maturity  Model.  The  employee 
handbook  is  structured  based  on  the  P-CMM  key  process  areas.  The  P-CMM  Process  Im¬ 
provement  Proposal  (PPBP)  mechanism  modeled  after  our  institutionalized  software  Process 
Improvement  Proposal  mechanism  is  the  vehicle  for  suggesting  improvements. 

4.2  People  Management  Processes 

The  Balanced  Scorecard  provides  a  link  from  the  strategic  objectives  to  the  position  objec¬ 
tives.  Figure  21  depicts  how  the  position  objectives  and  position  description  are  merged,  with 
the  outcome  being  the  self-evaluation  and  feedback  instrument  used  for  performance  man¬ 
agement. 
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Figure  21:  People  Management  System 

There  are  four  outcomes  from  the  self-evaluation  and  feedback  process: 

1.  Individual  improvement  goals 

2.  Summary  rating 

3.  Individual  training  goals 

4.  Individual  broadening  assignment  and  activities  goals 

Upon  completion  of  the  self-evaluation  form  by  the  employee  of  his  or  her  current  position 
accountabilities  and  professionalism  characteristics,  individual  improvement  goals  are  identi¬ 
fied  and  documented  in  a  discussion  between  the  employee  and  his  or  her  supervisor.  The 
summary  rating  is  one  criteria  used  for  compensation  increases. 


The  training  director  incorporates  the  individual  training  goals  into  a  plan  for  the  organiza¬ 
tion.  The  individual  broadening  assignments  and  activities  are  used  in  preparing  a  report  that 
managers  and  leaders  of  groups  such  as  SEPG  and  SQA  use  for  staffing  both  part-time  and 
full-time  functions. 


The  links  from  the  Balanced  Scorecard  to  an  individual’s  goals  for  improvement,  training, 
and  broadening  assignments  and  activities  help  to  align  the  individual’s  career  plans  with  the 
company  goals. 
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4.2.1  Role  of  Software  Engineer  In  Software  Process 
Improvement 

The  role  of  the  software  engineer  in  software  process  improvement  is  defined  by  the  account¬ 
abilities  of  the  position.  These  accountabilities  are  measured  in  the  performance  evaluation 
and  feedback  process.  Accountabilities  include: 

•  Ensure  use  of  an  approved  tailoring  of  the  PSP  in  software  development. 

•  Acquire  and  demonstrate  an  in-depth  understanding  of  the  SEI CMM. 

•  Ensure  use  of  the  project  software  engineering  process  that  is  an  approved  tailoring  of  the 
AIS  defined  process. 

•  Ensure  the  highest  possible  quality  in  the  development  of  a  component  by  removing  a 
targeted  percentage  of  defects  before  compile  and  test. 

•  Ensure  accurate  and  timely  submittal  of  size,  effort,  schedule,  and  defect  data  to  the 
Project  Manager. 

•  Ensure  improvement  of  the  AIS  defined  process  by  submitting  PIPs. 

4.2.2  Role  of  Project  Manager  in  Software  Process 
Improvement 

The  role  of  the  Project  Manager  in  software  process  improvement  is  defined  by  the  account¬ 
abilities  of  the  position.  These  accountabilities  are  measured  in  the  performance  evaluation 
and  feedback  process.  Accountabilities  include: 

•  Ensure  use  of  an  approved  tailoring  of  the  PSP  in  software  development. 

•  Acquire  and  demonstrate  an  in-depth  understanding  of  the  SEI  CMM. 

•  Ensure  that  project  processes  are  planned,  tracked,  and  analyzed  according  to  their 
defined  process  that  is  an  approved  tailoring  of  the  AIS  organization  defined  process. 

•  Ensure  project  staff  awareness  and  understanding  of  the  project’s  software  engineering 
processes  and  product  specifications. 

•  Ensure  accuracy  and  timeliness  of  data  collected  for  project  and  process  management. 

•  Ensure  that  project  post  mortems  are  conducted  and  result  in  the  submittal  of  PIPs  for  the 
improvement  of  the  AIS  defined  process. 

The  Development  Manager,  Process  Manager,  and  Training  Director  all  have  accountabilities 
similar  to  the  Project  Manager  that  are  tied  to  software  process  improvement. 

4.3  Long-Range  Plan 

A  Long-Range  Planning  Group  of  18  key  employees  was  established  in  September  1997.  The 
focus  of  this  group  was  to  establish  a  corporate  long-term  strategy  to  take  AIS  into  the  year 
2007.  Figure  22  iterates  the  purpose  of  the  long-range  plan.  Currently  there  are  five  teams 
working  on  implementing  the  critical  success  factors  (CSFs)  that  were  identified.  The  largest 
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team  is  working  on  Critical  Success  Factor  #3.  One  of  the  items  this  group  is  working  on  is  to 
define  processes  that  our  on-site  consultants  can  use  in  non-process  oriented  environments. 


/  Our  Purpose: 
Continuously  advance  the 
boundaries  of  quality. 


What  we  want  for 
our  future 


What  we  believe 


What  we  will  be 


Our  metrics 


What  we 
must  do 


Figure  22:  Long-Range  Plan 
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Appendix  A:  The  AIS  Approach  to  the 

Balanced  Scorecard 


AIS  has  adapted  the  Balanced  Scorecard  approach  for  its  organization’s  objectives  and  needs. 
There  are  cause-and-effect  relationships  throughout  the  Balanced  Scorecard.  When  “Engi¬ 
neers  use  the  Personal  Software  Process,”  which  is  a  Learning  and  Growth  performance 
driver,  that  also  enables  “Components  with  target  percent  of  defects  removed  before  compile 
and  test.”  This  leads  to  “Components,  modules,  programs  with  zero  integration  test  defects” 
to  meet  the  Internal  Business  Process  strategic  objective  of  “Engineers  achieve  the  highest 
possible  quality  in  the  design,  code  phases  of  a  component,  module  or  program.”  Satisfying 
that  objective  drives  “Defect-free  deliveries”  and  “Customer  responses  indicating  value 
‘achieved’”  to  satisfy  the  strategic  objective  from  the  Customer  perspective  to  “Consistently 
meet  or  exceed  customer  expectations  for  defect-free  and  on-time  delivery.” 

The  Balanced  Scorecard  approach  ensures  that  CPI  benefits  will  continue  in  the  future  by 
demonstrating  management  commitment  to  CPI,  formulating  strategy  based  on  CPI,  commu¬ 
nicating  the  strategy  to  the  entire  organization,  and  aligning  employee  career  goals  with  or¬ 
ganization  goals. 

The  Balanced  Scorecard  demonstrates  the  company  management’s  commitment  to  continu¬ 
ous  process  improvement.  This  method  communicates  strategy  to  the  entire  organization, 
links  individual  accountabilities  to  strategic  objectives,  and  provides  a  method  against  which 
to  measure  progress. 
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The  following  matrix  illustrates  how  this  approach  is  used  by  AIS  as  a  guideline. 


Strategic  Objectives 

Strategic  Measurements 
Core  Outcomes 

Strategic  Measurements 
Performance  Drivers 

Financial 

Consistently  meet  or  exceed  share¬ 
holder  expectations  for 

-  revenue  growth 

-  profitability 

-  return  on  investment 

Employee  target  ratio  of  gross 
revenue  to  base  salary 

Projects’  profitability  target 

Increase  in  shareholder  equity 

Designated  expenses  target  reduc¬ 
tion  in  expense  to  revenue  ratio 

Customer 

Consistently  meet  or  exceed  cus¬ 
tomer  expectations  for 

-  defect  free  and  on-time  delivery 

-  value  for  products  and  services 

-  achieving  time-to-market  goals 

Customer  responses  indicating 
value  “achieved” 

Statements  of  Work  lost  due  to  not 
meeting  customer  time  to  market 
goals 

Defect-free  deliveries 

On-time  or  ahead-of-schedule  de¬ 
liveries 

Employee 

Consistently  meet  or  exceed  em¬ 
ployee  expectations  for 

-  training 

-  compensation 

-  communication 

-  work  environment 

-  performance  management 

-  career  development 

Employee  responses  and  assess¬ 
ment  indicating  P-CMM  Repeat- 
able  level  key  process  areas  fully 
satisfied 

Disciplined,  repeatable,  and  stable 
work  force  practices  documented 

Internal  Business  Process 

Projects  achieve  predictable  results 
for  effort,  schedule,  and  defects 
within  known  range  of  AIS  organi¬ 
zation  defined  process  capability 

Engineers  achieve  the  highest  pos¬ 
sible  quality  in  the  design,  code 
phases  of  a  component,  module  or 
program 

AIS  organization  defined  process  is 
continuously  improved 

Projects  with  actual  effort  and 
schedule  less  than  committed  ef¬ 
fort  and  schedule 

Components,  modules,  programs 
with  zero  integration  test  defects 

New  products  or  product  en¬ 
hancements  with  documented 
quality  better  than  its  predecessor 

Projects  planned  and  managed 
according  to  their  defined  process 
which  is  an  approved  tailoring  of 
the  AIS  organization  defined  proc¬ 
ess 

Components  with  target  percent  of 
defects  removed  before  compile 
and  test 

Process  Improvement  Proposals 
submitted  and  implemented 

Learning  and  Growth 

Investment  in  people,  process,  and 
technology  enables  achievement  of 
customer,  employee,  and  share¬ 
holder  satisfaction  goals 

Engineers  achieving  training  goals 

Engineers  align  their  career  goals 
with  company  goals 

Engineers  improve  productivity 
continuously 

Engineers  acquire  new  skills 

Engineers  achieving  career  plans 

Engineers  use  the  Personal  Soft¬ 
ware  Process 

34 


CMU/SEI-99-TR-027 


References 


[Curtis  95] 

[Ebenau  94] 

[Humphrey  87] 

[Humphrey  89] 

[Humphrey  95] 

[Kaplan  96] 

[Paulk  93] 


Curtis,  Bill;  Hefley,  William  E.;  &  Miller,  Sally.  People  Capability 
Maturity  Model  (CMU/SEI-95-MM-002,  ADA  300822).  Pittsburgh, 
PA:  Software  Engineering  Institute,  Carnegie  Mellon  University, 
1995.  Available  WWW  <URL:  http://www.sei.cmu.edu/ 
publications/documents/95  .reports/95.mm.002.html>. 

Ebenau,  Robert  G  &  Strauss,  Susan  H.  Software  Inspection  Proc¬ 
ess.  AT&T,  1994. 

Humphrey,  W.  And  Sweet,  W.  Method  for  Assessing  the  Software 
Engineering  Capability  of  Contractors  (CMU/SEI-87-TR-23,  ADA 
187230).  Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie 
Mellon  University,  1987. 

Humphrey,  Watts.  Managing  the  Software  Process.  Reading,  MA: 
Addison-Wesley  Publishing  Company,  Inc.,  1989. 

Humphrey,  Watts.  A  Discipline  for  Software  Engineering.  Reading, 
MA:  Addison-Wesley  Publishing  Company,  Inc.,  1995. 

Kaplan,  Robert  S.  &  Norton,  David  P.  The  Balanced  Scorecard. 
President  and  Fellows  of  Harvard  College,  1996. 

Paulk,  Mark  C.;  Curtis,  Bill;  Chrissis,  Mary  Beth;  &  Weber,  Charles 
V.  Capability  Maturity  Model  for  Software,  Version  1.1  (CMU/SEI- 
93-TR-024,  ADA  263403).  Pittsburgh,  PA:  Software  Engineering 
Institute,  Carnegie  Mellon  University,  1993.  Available  WWW 
<URL:  http://www.sei.cmu.edu/publications/documents/93.reports/ 
93.tr.024.html>. 


CMU/SEI-99-TR-027 


35 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  main¬ 
taining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information ^'n^d'ng  sug¬ 
gestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and 
to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. - - — - 

1.  AGENCY  USE  ONLY  (LEAVE  BLANK)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVER  EC 

November  1999  cinoi 


TITLE  AND  SUBTITLE 

Software  Process  Improvement  Works! 

6.  AUTHOR(S) 

Pat  Ferguson,  Gloria  Leman,  Prasad  Perini,  Susan  Renner,  Girish  Seshagiri 

Y  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Software  Engineering  Institute 
Carnegie  Mellon  University 

Pittsburgh,  PA  15213 _ _ _  - 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

HQ  ESC/XPK 
5  Eglin  Street 

Hanscom  AFB,  MA  01731-2116 _ 


1 1 .  SUPPLEMENTARY  NOTES 


REPORT  TYPE  AND  DATES  COVERED 

Final _ 

FUNDING  NUMBERS 

C  —  F19628-95-C-0003 


PERFORMING  ORGANIZATION 
REPORT  NUMBER 
CMU/SEI-99-TR-027 


SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 
ESC-TR-99-026 


12^A  DISTRIBUTION/AVAILABILITY  STATEMENT  12b  DISTRIBUTION  CODE 

Unclassified/Unlimited,  DTIC,  NTIS _ _ _ 

ABSTRACT  (MAXIMUM  200  WORDS)  #  ^ 

The  Advanced  Information  Services  Inc.  (AIS)  Development  Group  was  motivated  by  business  needs  to  begin  its  Continuous  Process 
Improvement  (CPI)  initiative  in  1992.  We  chose  the  Software  Engineering  Institute  (SEI)  Capability  Maturity  Model®  (CMM®)  as  the  proc¬ 
ess  maturity  framework  [Paulk  93]  and  the  Institute  of  Electrical  and  Electronics  Engineers  (IEEE)  standards  as  the  guidelines  for  soft¬ 
ware  engineering. 

We  can  relate  the  changes  in  AIS  process  capability  to  three  different  eras.  The  first  was  the  time  before  1992,  during  which  AIS  did  not 
use  a  model  for  software  process  improvement  During  the  second  era,  1992  to  1995,  the  CMM  was  used.  Data  indicate  that  project 
schedule  and  effort  predictability  improved  with  the  Introduction  of  the  CMM  framework.  The  third  era  began  in  1995  when  AIS  piloted 
the  Personal  Software  Process8*  (PSP***).  With  the  adoption  of  the  PSP,  data  reflect  an  Increase  in  product  quality  and  improvements 
in  productivity  levels. 


14.  subject  terms  Balanced  Scorecard,  Capability  Maturity  Model  (CMM  ),  Personal  Software 


Process 


rocess  assessment,  process  improvement,  process  management 


15.  NUMBER  OF  PAGES 
40 

16.  Price  Code 


1 9.  SECURITY  CLASSIFICATION 

20.  LIMITATION  OF  ABSTRACT 

OF  ABSTRACT 

UNCLASSIFIED 

UL 

